Model combining and interaction for medical imaging

ABSTRACT

This disclosure relates to the combining and interaction of multiple artificial intelligence (AI) models for medical image analysis. An example method includes obtaining AI models from model providers and organizing them to form associations. In response to a user request, base models are selected and provided. Additional models are further selected to combine with the base models, and medical image analysis results are presented based on applying a combination of the models to target medical image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalApplication No. 63/187,676, filed May 12, 2021, which application ishereby incorporated by reference in its entirety.

BACKGROUND Technical Field This disclosure relates generally to medicalimaging applications, and, more particularly, to the combining andinteraction of multiple artificial intelligence (AI) models for medicalimage analysis. Description of the Related Art

Medical images usually consist of two-dimensional images,three-dimensional images, or reconstructed fused images generatedthrough imaging equipment utilizing modern nuclear medicine techniques,for example, positron emission tomography (PET), computed tomography(CT), magnetic resonance imaging (MRI), functional MM (fMRI), X-ray,mammography, tomosynthesis, ultrasound or other modalities. Medicalimages may be viewed by the patient or health professionals in thecourse of rendering diagnosis, treatment, or other health care.

AI models can be used to facilitate the analysis of medical images. Forexample, a model in machine learning is the output of a machine learningalgorithm run on data. The model is typically saved after running amachine learning algorithm on training data (e.g., medical images thatwere collected in the past) and represents the rules, numbers, and anyother algorithm-specific data structures required to make predictions.Illustratively, a linear regression algorithm results in a modelcomprised of a vector of coefficients with specific values, a decisiontree algorithm results in a model comprised of a tree of if-thenstatements with specific values, and neural network backpropagation orgradient descent algorithms can result in a model comprised of a graphstructure with vectors or matrices of weights with specific values.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The embodiments of this disclosure are illustrated by way of example andnot limitation in the figures of the accompanying drawings, in whichlike references indicate similar elements and in which:

FIG. 1 is a block diagram illustrating an example networked environmentfor one or more model platforms in accordance with at least someembodiments of the techniques described herein.

FIG. 2 is a flow diagram illustrating an example process for organizingand combining models in accordance with at least some embodiments of thetechniques described herein.

FIG. 3 is a flow diagram illustrating an example process for medicalimage analysis based on model combining, in accordance with at leastsome embodiments of the techniques described herein.

FIG. 4 is a block diagram illustrating elements of an example computingdevice utilized in accordance with at least some embodiments of thetechniques described herein.

FIGS. 5a and 5b show an example of user interfaces related to medicalimage analysis based on combining complementary models, in accordancewith at least some embodiments of the techniques described herein.

FIGS. 6a and 6b show another example of user interfaces related tomedical image analysis based on combining complementary models, inaccordance with at least some embodiments of the techniques describedherein.

FIGS. 7a and 7b show an example of user interfaces related to medicalimage analysis based on combining models of the same or similar type, inaccordance with at least some embodiments of the techniques describedherein.

FIGS. 8a and 8b show another example of user interfaces related tomedical image analysis based on combining models of the same or similartype, in accordance with at least some embodiments of the techniquesdescribed herein.

FIG. 9 shows an example of user interfaces related to a configurableworkflow for medical image analysis that enables individual models belinked together, in accordance with at least some embodiments of thetechniques described herein.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description that follows.

DETAILED DESCRIPTION

The following description, along with the accompanying drawings, setsforth certain specific details in order to provide a thoroughunderstanding of various disclosed embodiments. However, one skilled inthe relevant art will recognize that the disclosed embodiments may bepracticed in various combinations, without one or more of these specificdetails, or with other methods, components, devices, materials, etc. Inother instances, well-known structures or components that are associatedwith the environment of the present disclosure, including but notlimited to the communication systems and networks and the environment,have not been shown or described in order to avoid unnecessarilyobscuring descriptions of the embodiments. Additionally, the variousembodiments may be methods, systems, media, or devices. Accordingly, thevarious embodiments may combine software and hardware aspects.

Throughout the specification, claims, and drawings, the following termstake the meaning explicitly associated herein, unless the contextclearly dictates otherwise. The term “herein” refers to thespecification, claims, and drawings associated with the currentapplication. The phrases “in one embodiment,” “in another embodiment,”“in various embodiments,” “in some embodiments,” “in other embodiments,”and other variations thereof refer to one or more features, structures,functions, limitations, or characteristics of the present disclosure,and are not limited to the same or different embodiments unless thecontext clearly dictates otherwise. As used herein, the term “or” is aninclusive “or” operator, and is equivalent to the phrases “A or B, orboth” or “A or B or C, or any combination thereof,” and lists withadditional elements are similarly treated. The term “based on” is notexclusive and allows for being based on additional features, functions,aspects, or limitations not described, unless the context clearlydictates otherwise. In addition, throughout the specification, themeaning of “a,” “an,” and “the” include singular and plural references.

References to the term “set” (e.g., “a set of items”), as used herein,unless otherwise noted or contradicted by context, is to be construed asa nonempty collection comprising one or more members or instances.

References to the term “subset” (e.g., “a subset of the set of items”),as used herein, unless otherwise noted or contradicted by context, is tobe construed as a nonempty collection comprising one or more members orinstances of a set or plurality of members or instances.

Moreover, the term “subset,” as used herein, refers to a proper subset,which is a collection of one or more members or instances that arecollectively smaller in number than the set or plurality of which thesubset is drawn. For instance, a subset of a set of ten items will haveless than ten items and at least one item.

Various applications, methods and systems are disclosed herein toprovide improved communication, integration, utilization, andcollaboration for AI based medical imaging environments. The describedembodiments provide a digital platform that enables various combinationsand interactions of AI models, to achieve innovative and effectivemedical imaging analysis that is far more than the sum of individual AImodel outcomes. For example, the platform facilitates combiningcomplementary models to generate AI-enhanced layer(s) over image data,combining models of the same or similar type to produce and present“average” or otherwise enhanced results, and generating and processingconfigurable workflows where models can be chained or otherwise linkedinto a logical flow. Further, the platform generates various novel userinterfaces or instructions for such interfaces to be presented remotely(e.g., on user devices). With these user interfaces, a user can control,guide, or otherwise interact with unified AI model processing andoutcome presentation.

In some embodiments, the user interfaces include a panel view, whichenables combinations of complementary models (e.g., models that aredesigned to generate different types of analysis results) to generate anAI-enhanced layer atop or beside image data. For example, the userinterface can present bounding boxes for object detection andlocalization, graded heat maps corresponding to clinical metrics, orother image overlay enhancements that are generated based on thecomplementary model results. Additionally, clinically relevant metricsand statistics (e.g. likelihood of lesion malignancy and an accompanyingconfidence score) can be presented on the side of the medical image. Theuser can toggle these overlay enhancements and analyses on or off.

In some embodiments, the user interfaces include a harmonized view,which enables combinations of models of the same or similar type (e.g.,models that provide different values for the same or overlappingparameters) to produce enhanced outputs (e.g., by using averages,intersections, unions, or the like). For example, given a chest CTimage, three lesion detection models can be combined to createslice-level annotation masks visible only in the regions of an imagewhere there is concordance between all three, and the odds of therebeing a malignant lesion requiring medical intervention can be aweighted average percent from all three models.

In some embodiments, the user interfaces include a workflow view whichenables individual models to be linked together to feed one or moremodels' outputs to other models as inputs. Illustratively, a user can,within the same view of the image, search for additional models to applyto the image. The search can be done with text input, by viewing popularmodels, or with other automatically applied filters that take intoaccount the type of image(s) being viewed. Once the user selectsadditional models, the user interface enables the user to “pull” theminto the workflow, which can be a chain, a tree, a lattice, or otherhierarchies to collectively produce and render medical imaging analysisresults based on the current images.

In some embodiments, the various configurations for combining models andgenerating their corresponding user interfaces can be saved and sharedbetween users. For example, a radiation oncologist may save athree-model “panel” to enhance and make more efficient his brainmetastases segmentation workflow. The three models can include twomodels that volumetrically segment the mets and one model that providesa prognostic score (e.g. probability of patient survival over the nexttwo years). This three-model combination can be shared with colleaguesand research collaborators, and applied to medical image series thatmeet criteria specific to the combination (e.g., as defined by a rangeof acceptable image header values).

FIG. 1 is a block diagram illustrating an example networked environment100 for one or more model platforms in accordance with at least someembodiments of the techniques described herein. The exemplary networkedenvironment 100 includes one or more model platforms 118, one or moremodel providers 128, and one or more user devices 138, which areinterconnected with one another via at least some part of connections108.

In the depicted exemplary networked environment 100, the connections 108may comprise one or more computer networks, one or more wired orwireless networks, satellite transmission media, one or more cellularnetworks, or some combination thereof. The connections 108 may include apublicly accessible network of linked networks, possibly operated byvarious distinct parties, such as the Internet. The connections 108 mayinclude other network types, such as one or more private networks (e.g.,corporate or university networks that are wholly or partiallyinaccessible to non-privileged users), and may include combinationsthereof, such that (for example) one or more of the private networkshave access to or from one or more of the public networks. Furthermore,the connections 108 may include various types of wired or wirelessnetworks in various situations, including satellite transmission. Inaddition, the connections 108 may include one or more communicationinterfaces to individual entities in the networked environment 100,various other mobile devices, computing devices and media devices,including but not limited to, radio frequency (RF) transceivers,cellular communication interfaces and antennas, USB interfaces, portsand connections (e.g., USB Type-A, USB Type-B, USB Type-C (or USB-C),USB mini A, USB mini B, USB micro A, USB micro C), other RF transceivers(e.g., infrared transceivers, Zigbee® network connection interfacesbased on the IEEE 802.15.4 specification, Z-Wave® connection interfaces,wireless Ethernet (“Wi-Fi”) interfaces, short range wireless (e.g.,Bluetooth®) interfaces and the like.

In various embodiments, examples of a user device 138 include, but arenot limited to, one or a combination of the following: a “computer,”“mobile device,” “tablet computer,” “smart phone,” “handheld computer,”or “workstation,” etc. The user device(s) 138 may be any suitablecomputing device or electronic equipment that is, e.g., operable tocommunicate with the model platform(s) 118, and to interact with user(s)for utilizing AI or other computational models that are contributed bythe model provider(s) and hosted on the model platform(s) 118.

In various embodiments, individual model platforms 118 and modelproviders 128 can be implemented in software or hardware form on one ormore computing devices including a “computer,” “mobile device,” “tabletcomputer,” “smart phone,” “handheld computer,” or “workstation,” etc.The model platform(s) 118 can perform model intake, model hosting, modelgrouping or association, model training, model execution, candidatemodel selection, model combining, model performance monitoring andfeedback, or other model-related functions described herein. The modelprovider(s) 128 can provide AI or other computational models (e.g., thatare designed or trained by developers) and associated model metadata, aswell as processing model performance feedback, or other model-relatedfunctions. A model provider 128 can be a user device 138 or a modelplatform 118 in accordance with at least some implementation of thepresently disclosed technology.

Data communications among entities of the networked environment 100 canbe encrypted. Related encryption and decryption may be performed asapplicable according to one or more of any number of currently availableor subsequently developed encryption methods, processes, standards,protocols, or algorithms, including but not limited to: encryptionprocesses utilizing a public-key infrastructure (PKI), encryptionprocesses utilizing digital certificates, the Data Encryption Standard(DES), the Advanced Encryption Standard (AES 128, AES 192, AES 256,etc.), the Common Scrambling Algorithm (CSA), encryption algorithmssupporting Transport Layer Security 1.0, 1.1, or 1.2, encryptionalgorithms supporting the Extended Validation (EV) Certificate, etc.

The above description of the exemplary networked environment 100 and thevarious service providers, systems, networks, and devices therein isintended as a broad, non-limiting overview of an exemplary environmentin which various embodiments of the facility may be implemented. FIG. 1illustrates just one example of an operating environment, and thevarious embodiments discussed herein are not limited to suchenvironments. In particular, the networked environment 100 may containother devices, systems or media not specifically described herein.

FIG. 2 is a flow diagram illustrating an example process 200 fororganizing and combining models in accordance with at least someembodiments of the techniques described herein. Illustratively, theprocess 200 can be implemented by a model platform 118, and in someinstances, in communication with one or more model providers 128 anduser devices 138.

At block 202, the process 200 includes obtaining one or more AI or othercomputational models from the model provider(s) 128. The model platform118 can implement an interface (e.g., via HTTP, FTP, or other applicableprotocols) where model provider(s) 128 can upload models. In someembodiments, training data, testing data, or metadata associated withthe model(s) can also be uploaded. The metadata can include descriptionsabout the input, output, structure, or parameters of the model, forexample. The metadata can conform to pre-defined grammar, keywords ortextural structure, so that it can be quickly parsed by the modelplatform 118 in accordance with certain pre-defined rules. In someembodiments, the metadata does not conform to any defined grammar orstructure, and can include freeform texts. In these cases, the modelplatform 118 can implement applicable natural language processingtechniques to analyze the content of the metadata. In some embodiments,the model platform 118 can analyze the uploaded model itself anddetermine its input, output, structure, or parameters, without accessingmetadata.

At block 204, the process 200 includes organizing the models (e.g.,previously obtained and newly obtained) to form associations among themand thereby facilitate their retrieval. Illustratively, the modelplatform 118 can organize the models into groups or multiple levels ofgroups and subgroups. As non-limiting examples, grouping criteria can bethe similarity of input or output of models, overlaps between a model'soutput and another model's input, or structural similarity of models.Individual groups (or subgroups) may or may not include model(s) incommon, and may reference one another to provide further flexibility inassociating the models with one another.

At block 206, the process 200 includes providing one or more base modelsin response to a user request. In some embodiments, the user request isgenerated based on a user's interaction with one or more user interfacesdescribed herein in accordance with the presently disclosed technology.The user request can be transmitted from a user device 138 to the modelplatform 118. The user request can indicate an analysis purpose,context, applicable medical data, model structure, model input andoutput, or performance requirement. Based on the user request, the modelplatform 118 searches the organized models, selects the one or more basemodels, and provides them to the user device 138 (e.g., via the one ormore user interfaces).

At block 208, the process 200 includes providing one or more additionalmodels for model combining. As described above, model combining caninclude combining complementary models (e.g., to generate AI-enhancedlayer(s) over image data), combining models of the same or similar type(e.g., to produce and present enhanced, integrated results), orgenerating and processing configurable workflow(s) (e.g., so that modelscan be linked into a logical flow to produce results as a whole).

In some embodiments, complementary models are models designed to receivethe same or overlapping input features and to generate different outputfeatures (e.g., for different analyses or purposes). In someembodiments, models of the same or similar type are different modelsdesigned to generate the same or overlapping output features (e.g., forthe same or similar analyses or purposes). Models of the same or similartype may be structurally different, having different values for theirinternal parameters, or designed to receive different input features. Insome embodiments, models that are linkable to form a configurableworkflow have compatibility between their inputs and outputs. Forexample, if model A′s output features match model B′s input features,then model A can be linked to model B. Further, multiple models' outputscan be combined (e.g., their output features or subsets thereof areselectively concatenated) to match another model's input, a singlemodel's output can be selectively subdivided to match multiple othermodels' inputs, and multiple models' outputs can be selectively combinedor subdivided into different feature combinations to match multipleother model's inputs.

Based on the base model(s) or in response to additional user request(s),the model platform 118 can search the organized models in accordancewith the models' associations, and identify candidate models forcombining with the base model(s). In some embodiments, the modelplatform 118 can further generate “dummy,” “padding,” or other neutralfeatures to supplement certain models' input or output, and therebyfacilitating the combining of the models.

At block 210, the process 200 includes executing the combination ofmodels and presenting analysis results. As described above, the modelplatform 118 can generate various user interfaces or instructions forsuch interfaces to be presented remotely (e.g., on user devices via abrowser or app). With these user interfaces, a user can control, guide,or otherwise interact with unified AI model processing and outcomepresentation. For example, the user can select from the candidate modelsto combine with the base model(s) or previously selected model(s),select medical data for model training, re-training, or testing,customize various views associated with the model combining and resultpresentation, provide feedback on performance of individual models ormodel combination(s), save the model combination or user interfacesettings for sharing with other users.

FIG. 3 is a flow diagram illustrating an example process 300 for medicalimage analysis based on model combining, in accordance with at leastsome embodiments of the techniques described herein. Illustratively, theprocess 300 can be implemented by a user device 138 in communicationwith one or more model platforms 118.

At block 302, the process 300 includes requesting one or more basemodels for medical image analysis. In some embodiments, the user device138 generates a user request based on a user's interaction with one ormore user interfaces described herein in accordance with the presentlydisclosed technology. The user request can be transmitted from the userdevice 138 to the model platform 118. In some embodiments, the userrequest specifies the base model(s). For example, the user selects thebase model(s) by browsing through various models organized andsearchable via applicable user interfaces, and the user device 138generates the user request including identifiers corresponding to theselected base model(s). In some embodiments, the user request canindicate an analysis purpose, context, applicable medical data, modelstructure, model input and output, or performance requirement. Based onthe user request, the model platform 118 searches the organized models,selects the one or more base models, and provides them to the userdevice 138 (e.g., via the one or more user interfaces).

At block 304, the process 300 includes selecting one or more additionalmodels for model combining. As described above, model combining caninclude combining complementary models (e.g., to generate AI-enhancedlayer(s) over image data), combining models of the same or similar type(e.g., to produce and present enhanced, integrated results), orgenerating and processing configurable workflow(s) (e.g., so that modelscan be linked into a logical flow to produce results as a whole).

In some embodiments, the user device 138 generates additional userrequest(s) based on the user's interaction with one or more userinterfaces described herein in accordance with the presently disclosedtechnology. The additional user request(s) can specify the additionalmodel(s) for combining. For example, the user selects the additionalmodel(s) associated with the base model(s) or previously selectedmodel(s), by browsing through various models organized and searchablevia applicable user interfaces, and the user device 138 generates theadditional user request(s) including identifiers corresponding to theselected additional model(s). In some embodiments, the additional userrequest(s) can indicate an analysis purpose, context, applicable medicaldata, model structure, model input and output, or performancerequirement for the additional models.

Based on the base model(s) or in response to additional user request(s),the model platform 118 can search the organized models in accordancewith their associations, and identify candidate models for combiningwith the base model(s). In some embodiments, the model platform 118 canfurther generate “dummy,” “padding,” or other neutral features tosupplement certain model's input or output, and thereby facilitating thecombining of the models.

At block 306, the process 300 includes interacting with model combiningand analysis results presentation. As described above, the modelplatform 118 can generate various user interfaces or instructions forsuch interfaces to be presented remotely (e.g., on user devices). Withthese user interfaces, a user can control, guide, or otherwise interactwith unified AI model processing and outcome presentation. For example,the user can select from the candidate models to combine with the basemodel(s) or previously selected model(s), select medical data for modeltraining, re-training, or testing, customize various views associatedwith the model combining and result presentation, provide feedback onperformance of individual models or model combination(s), save the modelcombination or user interface settings for sharing with other users.

FIGS. 5a and 5b show an example of user interfaces related to medicalimage analysis based on combining complementary models, in accordancewith at least some embodiments of the techniques described herein. Asdescribed above, the user interfaces can enable combinations ofcomplementary models to generate AI-enhanced layer(s) atop or besideimage data. The user interfaces can present bounding boxes 504 forobject detection and localization, graded heat maps 502 corresponding toclinical metrics, or other image overlay enhancements that are generatedbased on the complementary model results. Additionally, clinicallyrelevant metrics and statistics 506 a, 506 b can be presented on theside, top, or bottom of the medical image. The user can toggle theseoverlay enhancements and analyses on or off.

In the example shown in FIGS. 5a and 5b , multiple complementary modelsare combined for detecting concurrent lesions on a Chest X-ray study.Illustratively, three AI models from Milvue, Vuno and AiDA are appliedto medical image data of the Chest X-ray study. Vuno, Milvue and AiDAmodels returned findings of pneumonia related opacities, and Milvue andAiDA models found a lung nodule. More specifically in FIG. 5a , resultsfrom the Vuno and AiDA models show pneumonia probability maps on a ChestX-ray, and in FIG. 5b , results from the Milvue model include a detectedlung nodule in addition to the opacities.

FIGS. 6a and 6b show another example of user interfaces related tomedical image analysis based on combining complementary models, inaccordance with at least some embodiments of the techniques describedherein. The user interfaces facilitate running multiple AI models onChest CT studies and thereby enable a user (e.g., a radiologist) to lookfor multiple findings on a CT scan. As shown in FIG. 6a , the Altroxmodel is utilized to segment COVID opacities and quantify the ratio ofthe pneumonia burden, while the AiDx COPD model is utilized to evaluateemphysema and air trapping on the same medical imaging data. As shown inFIG. 6b , the Arterys Lung Malignancy Score model is utilized to detect,segment, and provide the likelihood of malignancy on the same medicalimaging data.

FIGS. 7a and 7b show an example of user interfaces related to medicalimage analysis based on combining models of the same or similar type, inaccordance with at least some embodiments of the techniques describedherein. As shown in FIG. 7a , two COVID segmentation models (e.g., PingAN and Altrox models) are applied on the same Chest CT imaging, and theygenerate respective COVID lesion masks (shown in two different shades ofcolor) that are concurrently overlaid onto the underlying medical image.This allows a user to get different reads on the estimation of findings,and serves as a basis for the clinical validation of the AI models.Alternatively or in addition, the user interfaces related to combiningmodels of the same or similar type facilitate the quality controlprocess of AI findings, and enable combining results to improve theconfidence in the AI findings. As shown in FIG. 7b , the COVID lesionmasks generated from the two models are combined (e.g., by a unionoperation) to show a unified mask in the same shade of color. In variousembodiments, the findings generated by different models can be combinedin different ways and presented to the user. For example, intersection,union, averaging, weighted averaging, probabilistic sampling,combination of the same or the like can be applied to the findings.

FIGS. 8a and 8b show another example of user interfaces related tomedical image analysis based on combining models of the same or similartype, in accordance with at least some embodiments of the techniquesdescribed herein. As shown in FIG. 8a , three pneumonia detection models(e.g., Vuno, AiDx, and RSNA2018) are applied on the same Chest X-rayimaging, and they generate respective suspected areas of pneumonia(shown as a segmentation mask, a probability mask, and a bounding box)that are concurrently overlaid onto the underlying medical image. Asshown in FIG. 8b , the findings from the three models are combined(e.g., by determining an overlapping area among the three results) toshow a unified, overlap mask overlaid onto the underlying medical image.

FIG. 9 shows an example of user interfaces related to a configurableworkflow for medical image analysis that enables individual models belinked together, in accordance with at least some embodiments of thetechniques described herein. As discussed above, using such userinterfaces, a user can, within the same view of the medical image(s),search for various models to apply to the image. The search can be donewith text input, by viewing popular models, or with other automaticallyapplied filters that take into account the type of image(s) beingviewed. The user interfaces enable the user to integrate the selectedmodels into the workflow, which can be a chain, a tree, a lattice, orother hierarchies to collectively produce and render medical imaginganalysis results based on the current images. As shown in FIG. 9, asequence of 4 AI models are used to read images in a Chest CT study forpurposes of identifying pathologies: a nodule detection model, a nodule3D segmentation model, a classification model providing the likelihoodof nodule malignancy, and a lobe segmentation model providing theestimation of emphysema and air trapped in the lungs.

FIG. 4 is a block diagram illustrating elements of an example computingdevice 400 utilized in accordance with at least some embodiments of thetechniques described herein. Illustratively, the computing device 400corresponds to a model platform 118, model provider 128, user device138, or at least a part thereof.

In some embodiments, one or more general purpose or special purposecomputing systems or devices may be used to implement the computingdevice 400. In addition, in some embodiments, the computing device 400may comprise one or more distinct computing systems or devices, and mayspan distributed locations. Furthermore, each block shown in FIG. 4 mayrepresent one or more such blocks as appropriate to a specificembodiment or may be combined with other blocks. Also, the model-relatedmanager 422 may be implemented in software, hardware, firmware, or insome combination to achieve the capabilities described herein.

As shown, the computing device 400 comprises a non-transitory computermemory (“memory”) 401, a display 402 (including, but not limited to alight emitting diode (LED) panel, cathode ray tube (CRT) display, liquidcrystal display (LCD), touch screen display, projector, etc.), one ormore Central Processing Units (“CPU”) or other processors 403,Input/Output (“I/O”) devices 404 (e.g., keyboard, mouse, RF or infraredreceiver, universal serial bus (USB) ports, High-Definition MultimediaInterface (HDMI) ports, other communication ports, and the like), othercomputer-readable media 405, and network connections 406. Themodel-related manager 422 is shown residing in memory 401. In otherembodiments, some portion of the contents and some, or all, of thecomponents of the model-related manager 422 may be stored on ortransmitted over the other computer-readable media 405. The componentsof the computing device 400 and model-related manager 422 can execute onone or more CPUs 403 and implement applicable functions describedherein. In some embodiments, the model-related manager 422 may operateas, be part of, or work in conjunction or cooperation with othersoftware applications stored in memory 401 or on various other computingdevices. In some embodiments, the model-related manager 422 alsofacilitates communication with peripheral devices via the I/O devices404, or with another device or system via the network connections 406.

The one or more model-related modules 424 is configured to performactions related, directly or indirectly, to AI or other computationalmodel(s). In some embodiments, the model-related module(s) 424 stores,retrieves, or otherwise accesses at least some model-related data onsome portion of the model-related data storage 416 or other data storageinternal or external to the computing device 400.

Other code or programs 430 (e.g., further data processing modules, aprogram guide manager module, a Web server, and the like), andpotentially other data repositories, such as data repository 420 forstoring other data, may also reside in the memory 401, and can executeon one or more CPUs 403. Of note, one or more of the components in FIG.4 may or may not be present in any specific implementation. For example,some embodiments may not provide other computer readable media 405 or adisplay 402.

In some embodiments, the computing device 400 and model-related manager422 include API(s) that provides programmatic access to add, remove, orchange one or more functions of the computing device 400. In someembodiments, components/modules of the computing device 400 andmodel-related manager 422 are implemented using standard programmingtechniques. For example, the model-related manager 222 may beimplemented as an executable running on the CPU 403, along with one ormore static or dynamic libraries. In other embodiments, the computingdevice 400 and model-related manager 422 may be implemented asinstructions processed by a virtual machine that executes as one of theother programs 430. In general, a range of programming languages knownin the art may be employed for implementing such example embodiments,including representative implementations of various programming languageparadigms, including but not limited to, object-oriented (e.g., Java,C++, C#, Visual Basic.NET, Smalltalk, and the like), functional (e.g.,ML, Lisp, Scheme, and the like), procedural (e.g., C, Pascal, Ada,Modula, and the like), scripting (e.g., Perl, Ruby, Python, JavaScript,VBScript, and the like), or declarative (e.g., SQL, Prolog, and thelike).

In a software or firmware implementation, instructions stored in amemory configure, when executed, one or more processors of the computingdevice 400 to perform the functions of the model-related manager 422. Insome embodiments, instructions cause the CPU 403 or some otherprocessor, such as an I/O controller/processor, to perform at least somefunctions described herein.

The embodiments described above may also use well-known or othersynchronous or asynchronous client-server computing techniques. However,the various components may be implemented using more monolithicprogramming techniques as well, for example, as an executable running ona single CPU computer system, or alternatively decomposed using avariety of structuring techniques known in the art, including but notlimited to, multiprogramming, multithreading, client-server, orpeer-to-peer, running on one or more computer systems each having one ormore CPUs or other processors. Some embodiments may execute concurrentlyand asynchronously, and communicate using message passing techniques.Equivalent synchronous embodiments are also supported by a model-relatedmanager 422 implementation. Also, other functions could be implementedor performed by each component/module, and in different orders, and bydifferent components/modules, yet still achieve the functions of thecomputing device 400 and model-related manager 422.

In addition, programming interfaces to the data stored as part of thecomputing device 400 and model-related manager 422, can be available bystandard mechanisms such as through C, C++, C#, and Java APIs; librariesfor accessing files, databases, or other data repositories; scriptinglanguages such as XML; or Web servers, FTP servers, NFS file servers, orother types of servers providing access to stored data. Themodel-related data storage 416 and data repository 420 may beimplemented as one or more database systems, file systems, or any othertechnique for storing such information, or any combination of the above,including implementations using distributed computing techniques.

Different configurations and locations of programs and data arecontemplated for use with techniques described herein. A variety ofdistributed computing techniques are appropriate for implementing thecomponents of the illustrated embodiments in a distributed mannerincluding but not limited to TCP/IP sockets, RPC, RMI, HTTP, and WebServices (XML-RPC, JAX-RPC, SOAP, and the like). Other variations arepossible. Other functionality could also be provided by eachcomponent/module, or existing functionality could be distributed amongstthe components/modules in different ways, yet still achieve thefunctions of the model-related manager 422.

Furthermore, in some embodiments, some or all of the components of thecomputing device 400 and model-related manager 422 may be implemented orprovided in other manners, such as at least partially in firmware orhardware, including, but not limited to one or more application-specificintegrated circuits (“ASICs”), standard integrated circuits, controllers(e.g., by executing appropriate instructions, and includingmicrocontrollers or embedded controllers), field-programmable gatearrays (“FPGAs”), complex programmable logic devices (“CPLDs”), and thelike. Some or all of the system components or data structures may alsobe stored as contents (e.g., as executable or other machine-readablesoftware instructions or structured data) on a computer-readable medium(e.g., as a hard disk; a memory; a computer network, cellular wirelessnetwork or other data transmission medium; or a portable media articleto be read by an appropriate drive or via an appropriate connection,such as a DVD or flash memory device) so as to enable or configure thecomputer-readable medium or one or more associated computing systems ordevices to execute or otherwise use, or provide the contents to perform,at least some of the described techniques.

The various embodiments described above can be combined to providefurther embodiments. All of the U.S. patents, U.S. patent applicationpublications, U.S. patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification or listed in the Application Data Sheet are incorporatedherein by reference, in their entirety. In cases where the presentpatent application conflicts with an application or other documentincorporated herein by reference, the present application controls.Aspects of the embodiments can be modified, if necessary to employconcepts of the various patents, applications and publications toprovide yet further embodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

1. A computer-implemented method for facilitating medical imageanalysis, comprising: obtaining a plurality of artificial intelligence(AI) models for medical image analysis from one or more model providers;organizing the plurality of models to form associations among themodels; in response to a user request received from a user device,providing one or more base models selected from the plurality of models;providing one or more additional models selected from the plurality ofmodels to combine with the one or more base models; and causingpresentation of medical image analysis results based, at least in part,on applying a combination of the one or more base models and the one ormore additional models to target medical image data.
 2. The method ofclaim 1, wherein organizing the plurality of models comprises organizingthe plurality of models into a hierarchy of groups based on groupingcriteria.
 3. The method of claim 2, wherein the grouping criteriaincludes at least one of a similarity of input between models, asimilarity of output between models, or an overlap between a model'soutput and another model's input.
 4. The method of claim 1, wherein theuser request indicates at least one of an analysis purpose, context,applicable medical data, model structure, model input or output, orperformance requirement.
 5. The method of claim 1, further comprisingselecting the one or more base models based, at least in part, on theassociations among the models.
 6. The method of claim 1, wherein the oneor more additional models and the one or more base models are designedto receive same or overlapping input features and to generate differentoutput features.
 7. The method of claim 1, wherein the one or moreadditional models and the one or more base models are designed togenerate same or overlapping output features.
 8. The method of claim 1,wherein the one or more additional models and the one or more basemodels are linkable to form a configurable workflow.
 9. The method ofclaim 8, wherein the configurable workflow includes at least one of achain, tree, or lattice structure to link models.
 10. The method ofclaim 1, wherein causing presentation of medical image analysis resultscomprises causing presentation of one or more user interfaces via theuser device.
 11. The method of claim 10, wherein causing presentation ofmedical image analysis results further comprises causing presentation ofimage overlay features corresponding to results from the combination ofthe one or more base models and the one or more additional models, viathe one or more user interfaces.
 12. One or more non-transitorycomputer-readable media collectively storing contents that, whenexecuted by one or more processors, cause the one or more processors toperform actions comprising: organizing a plurality of models for medicalimage analysis to form associations among the models; combining a subsetof the plurality of models to form a combination of models based, atleast in part, on a user request and the associations among the models;and causing presentation of medical image analysis results based, atleast in part, on applying the combination of models to target medicalimage data.
 13. The one or more non-transitory computer-readable mediaof claim 12, wherein causing presentation of medical image analysisresults comprises causing presentation of image overlay featuresconcurrently with one or more images of the target medical image data.14. The one or more non-transitory computer-readable media of claim 13,wherein the image overlay features include at least one of boundingboxes for object detection and localization, or graded heat mapscorresponding to clinical metrics.
 15. The one or more non-transitorycomputer-readable media of claim 13, wherein the image overlay featuresinclude a single feature that integrates results generated fromindividual models of the subset.
 16. The one or more non-transitorycomputer-readable media of claim 15, wherein the single feature isgenerated by at least one of intersection, union, averaging, weightedaveraging, or probabilistic sampling operation.
 17. A system,comprising: one or more processors; and non-transitory memory storingcontents that, when executed by the one or more processors, cause thesystem to: organize a plurality of models for medical image analysis toform associations among the models; combine a subset of the plurality ofmodels to form a combination of models based, at least in part, on auser request and the associations among the models; and causepresentation of medical image analysis results based, at least in part,on applying the combination of models to target medical image data. 18.The system of claim 17, wherein organizing the plurality of modelscomprises organizing the plurality of models into a hierarchy of groupsbased on grouping criteria.
 19. The system of claim 17, whereinindividual models of the subset are designed to receive same oroverlapping input features and to generate different output features.20. The system of claim 17, wherein individual models of the subset aredesigned to generate same or overlapping output features.
 21. The systemof claim 17, wherein individual models of the subset are linkable toform a workflow including at least one of a chain, tree, or latticestructure to link models.